Papers with word prediction
Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Models (2025.tacl-1)
Copied to clipboard
| Challenge: | a recent study revisits six core challenges that have influenced the evolution of Neural Machine Translation (NMT) domain mismatch, amount of parallel data, rare word prediction, translation of long sentences and sub-optimal beam search remain challenges in LLMs. |
| Approach: | They revisit core challenges that have acted as benchmarks for progress in NMT . they propose to revisit these challenges and offer insights into their relevance . |
| Outcome: | The proposed models significantly improve translation of sentences containing approximately 80 words, even translating documents up to 512 words. |
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2021.tacl-1)
Copied to clipboard
| Challenge: | Amnesic probing is a method that focuses on how information is being used, rather than on what information is encoded. |
| Approach: | They propose a method that focuses on how the information is being used rather than on what information is encoded. |
| Outcome: | The proposed method is based on a BERT dataset to ask questions that were not possible before . it shows that probing performance is not correlated to task importance . |
Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings (N18-2)
Copied to clipboard
| Challenge: | Existing word embedding models are limited by semantic resources, which are hard to obtain or annotate. |
| Approach: | They propose a directional skip-gram model that explicitly distinguishes between left and right contexts in word prediction. |
| Outcome: | The proposed model outperforms other models on different datasets in semantic and syntactic evaluations. |
On-Device Neural Language Model Based Word Prediction (C18-2)
Copied to clipboard
| Challenge: | Currently, on-device keyboards have limited memory and response time for word prediction . a proposed on-device neural language model based word prediction method is available for mobile devices . |
| Approach: | They propose an on-device neural language model based word prediction method that optimizes run-time memory and provides a real-time prediction environment. |
| Outcome: | The proposed model outperforms existing methods for word prediction in keystroke savings and word prediction rate and has been commercialized. |
Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment (2024.findings-naacl)
Copied to clipboard
| Challenge: | Current approaches to obtain cross-lingual sentence embeddings rely on pre-trained language models that implicitly align the contextual representations of similar units of sentences in different languages. |
| Approach: | They propose a framework that explicitly aligns words between English and eight low-resource languages by using off-the-shelf word alignment models. |
| Outcome: | The proposed framework improves on the bitext retrieval task and in high-resource languages. |
Attribute-aware Sequence Network for Review Summarization (D19-1)
Copied to clipboard
| Challenge: | Existing review summarization systems generate summary only based on review content and neglect the authors’ attributes (e.g., gender, age, and occupation). |
| Approach: | They propose an Attribute-aware Sequence Network (ASN) to take the aforementioned users’ characteristics into account by encoding their attributes over the words. |
| Outcome: | The proposed model outperforms existing systems on tripAtt and human evaluation by taking the authors' attributes into account and incorporating attribute embedding and word-using habits into word prediction. |
Evaluating Approaches to Personalizing Language Models (2020.lrec-1)
Copied to clipboard
| Challenge: | a large amount of text is not available for training a user-specific language model, which suggests a need to personalize language models with only a small amount of data. |
| Approach: | They propose three approaches to personalize a language model that was trained on a large background corpus using a relatively small amount of text from an individual user. |
| Outcome: | The proposed techniques outperform language model adaptation based on demographic factors. |
Exploring BERT’s Sensitivity to Lexical Cues using Tests from Semantic Priming (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Using English lexical stimuli, we find that BERT models show "priming" predicting a word with greater probability when the context includes a related word versus an unrelated one. |
| Approach: | They analyze a pre-trained BERT model with tests informed by semantic priming . they find that BERT too shows "priming" predicting a word with greater probability when context includes a related word versus an unrelated one. |
| Outcome: | The proposed model shows a tendency to be distracted by related prime words as context becomes more informative, and lower probability of related words. |
Online Infix Probability Computation for Probabilistic Finite Automata (P19-1)
Copied to clipboard
| Challenge: | Probabilistic finite automata (PFAs) are statistical language models used in natural language processing. |
| Approach: | They develop an asymptotic algorithm to compute the infix probabilities of each prefix of a string from streaming data. |
| Outcome: | The proposed algorithm improves the infix probabilities of a weighted automata from streaming data. |
On Entity Identification in Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work has shed light on the internal mechanisms of language models that can recall factual knowledge composed of entities and relations. |
| Approach: | They propose a framework analogous to clustering quality metrics to analyze the correspondence between entities and their mentions. |
| Outcome: | The proposed framework is analogous to clustering quality metrics. |